Skip to content

feat(low-code): add RateLimitedMultipleTokenAuthenticator component#1066

Draft
Daryna Ishchenko (darynaishchenko) wants to merge 8 commits into
mainfrom
devin/1783347505-rate-limited-multiple-token-authenticator
Draft

feat(low-code): add RateLimitedMultipleTokenAuthenticator component#1066
Daryna Ishchenko (darynaishchenko) wants to merge 8 commits into
mainfrom
devin/1783347505-rate-limited-multiple-token-authenticator

Conversation

@darynaishchenko

@darynaishchenko Daryna Ishchenko (darynaishchenko) commented Jul 6, 2026

Copy link
Copy Markdown
Contributor

Summary

Adds a new declarative authenticator, RateLimitedMultipleTokenAuthenticator, that generalizes source-github's MultipleTokenAuthenticatorWithRateLimiter so it can be reused by any connector during declarative migrations (context: https://github.com/airbytehq/airbyte-internal-issues/issues/16512).

Behavior (per outgoing request, via AuthBase.__call__):

  • Requests are classified into named quota pools (quotas) via HttpRequestRegexMatchers; a pool with no matchers is the default (generalizes GitHub's hardcoded "graphql" in path REST-vs-GraphQL split).
  • Each token tracks a local per-pool counter, seeded once per token from quota_status_source (e.g. GitHub GET /rate_limit) using remaining_path/reset_path/limit_path dpath extraction. Seeding is lazy (first request) and repeated after an exhaustion wait.
  • Active token's counter is decremented; on exhaustion the authenticator rotates through tokens; when all are drained it sleeps until the earliest reset (heartbeat logs, floored at MIN_EXHAUSTION_WAIT = 5s so stale reset timestamps can't busy-loop the quota endpoint, bounded by max_wait_time, else raises transient_error).
  • Proactive throttling: once every token drops below max(budget_min_reserve, budget_reserve_fraction * limit), delays of seconds_to_reset / total_remaining (capped at 10s) are injected for graceful degradation instead of a hard wall.

Thread safety (new vs. the source-github implementation, required for the concurrent CDK):

  • Token selection + decrement happen atomically under an RLock; _acquire_call returns the token chosen under the lock and the request header is built from that value, so a request is always signed with the token whose counter was decremented (no TOCTOU with concurrent rotation). All sleeps occur outside the lock.
  • Double-checked re-seed via a separate _refresh_lock so only one thread refreshes counters after an exhaustion wait.
  • ModelToComponentFactory.create_rate_limited_multiple_token_authenticator caches instances keyed on the model definition, so all streams share one counter set (same idea as the shared api_budget).

Example manifest usage (source-github):

authenticator:
  type: RateLimitedMultipleTokenAuthenticator
  tokens: "{{ config['credentials']['personal_access_token'] }}"  # comma-separated PATs, or a list
  auth_method: "token"
  quota_status_source:
    type: QuotaStatusSource
    url: "{{ config.get('api_url', 'https://api.github.com') }}/rate_limit"
  quotas:
    - type: TokenQuota
      name: rest
      remaining_path: ["resources", "core", "remaining"]
      reset_path: ["resources", "core", "reset"]
      limit_path: ["resources", "core", "limit"]
    - type: TokenQuota
      name: graphql
      remaining_path: ["resources", "graphql", "remaining"]
      reset_path: ["resources", "graphql", "reset"]
      matchers:
        - type: HttpRequestRegexMatcher
          url_path_pattern: "/graphql"

Changes:

  • declarative_component_schema.yaml: new RateLimitedMultipleTokenAuthenticator, QuotaStatusSource, TokenQuota definitions; added to HttpRequester.authenticator and SelectiveAuthenticator.authenticators unions (models regenerated; new classes applied on top of the checked-in file to avoid unrelated codegen churn).
  • airbyte_cdk/sources/declarative/auth/rate_limited_multiple_token.py: runtime implementation.
  • model_to_component_factory.py: create_rate_limited_multiple_token_authenticator with token-string splitting, interpolation, and shared-instance caching.
  • Unit tests: seeding/header injection, quota matching, rotation, exhaustion wait/raise, budget throttling, thread safety (no lost decrements across 200 threads; signed token matches decremented token under concurrent rotation), factory token parsing and instance sharing.

Requested by Daryna Ishchenko (darynaishchenko).

Link to Devin session: https://app.devin.ai/sessions/3ced8f6a11d44ed4abc517221294f767
Requested by: Daryna Ishchenko (@darynaishchenko)

… component

Co-Authored-By: Daryna Ishchenko <darina.ishchenko17@gmail.com>
@devin-ai-integration

Copy link
Copy Markdown
Contributor

🤖 Devin AI Engineer

I'll be helping with this pull request! Here's what you should know:

✅ I will automatically:

  • Address comments on this PR. Add '(aside)' to your comment to have me ignore it.
  • Look at CI failures and help fix them

Note: I can only respond to comments from users who have write access to this repository.

⚙️ Control Options:

  • Disable automatic comment, CI, and merge conflict monitoring

@github-actions

github-actions Bot commented Jul 6, 2026

Copy link
Copy Markdown

👋 Greetings, Airbyte Team Member!

Here are some helpful tips and reminders for your convenience.

💡 Show Tips and Tricks

Testing This CDK Version

You can test this version of the CDK using the following:

# Run the CLI from this branch:
uvx 'git+https://github.com/airbytehq/airbyte-python-cdk.git@devin/1783347505-rate-limited-multiple-token-authenticator#egg=airbyte-python-cdk[dev]' --help

# Update a connector to use the CDK from this branch ref:
cd airbyte-integrations/connectors/source-example
poe use-cdk-branch devin/1783347505-rate-limited-multiple-token-authenticator

PR Slash Commands

Airbyte Maintainers can execute the following slash commands on your PR:

  • /autofix - Fixes most formatting and linting issues
  • /poetry-lock - Updates poetry.lock file
  • /test - Runs connector tests with the updated CDK
  • /prerelease - Triggers a prerelease publish with default arguments
  • /poe build - Regenerate git-committed build artifacts, such as the pydantic models which are generated from the manifest JSON schema in YAML.
  • /poe <command> - Runs any poe command in the CDK environment
📚 Show Repo Guidance

Helpful Resources

📝 Edit this welcome message.

Co-Authored-By: Daryna Ishchenko <darina.ishchenko17@gmail.com>
@github-actions

github-actions Bot commented Jul 6, 2026

Copy link
Copy Markdown

PyTest Results (Fast)

4 139 tests  +16   4 127 ✅ +16   7m 57s ⏱️ +21s
    1 suites ± 0      12 💤 ± 0 
    1 files   ± 0       0 ❌ ± 0 

Results for commit f300579. ± Comparison against base commit 53785b6.

♻️ This comment has been updated with latest results.

@github-actions

github-actions Bot commented Jul 6, 2026

Copy link
Copy Markdown

PyTest Results (Full)

4 142 tests   4 130 ✅  12m 20s ⏱️
    1 suites     12 💤
    1 files        0 ❌

Results for commit f300579.

♻️ This comment has been updated with latest results.

devin-ai-integration Bot and others added 2 commits July 6, 2026 15:42
…on wait

Co-Authored-By: Daryna Ishchenko <darina.ishchenko17@gmail.com>
Co-Authored-By: Daryna Ishchenko <darina.ishchenko17@gmail.com>
devin-ai-integration Bot and others added 4 commits July 7, 2026 12:20
… already converted

Co-Authored-By: Daryna Ishchenko <darina.ishchenko17@gmail.com>
Co-Authored-By: Daryna Ishchenko <darina.ishchenko17@gmail.com>
Co-Authored-By: Daryna Ishchenko <darina.ishchenko17@gmail.com>
Co-Authored-By: Daryna Ishchenko <darina.ishchenko17@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant